Add vLLM AFD Plugin blog post - #263
Conversation
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
d93dc71 to
0cb8f55
Compare
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c83b2854ab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - ecosystem | ||
| --- | ||
|
|
||
| We are excited to introduce [**vLLM AFD Plugin**](https://github.com/vllm-project/afd-plugin), an experimental external plugin that brings **Attention–FFN Disaggregation (AFD)** to vLLM. |
There was a problem hiding this comment.
This announcement sends readers to https://github.com/vllm-project/afd-plugin for the plugin, install steps, recipes, docs, and issues, but that repository is not publicly reachable right now; exact URL searches do not surface the repo and a direct browser open returns GitHub's missing-repository page. Once this post is published, users will hit broken install/documentation links unless the repo is made public or these URLs are updated to the actual public location.
Useful? React with 👍 / 👎.
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
|
|
||
| The next phase of development will focus on: | ||
|
|
||
| * **Broader vLLM compatibility and upstream alignment:** track newer vLLM and vLLM-Ascend releases, evaluate model runner v2, keep compatibility patches minimal, and contribute generally useful abstractions upstream as they mature. |
There was a problem hiding this comment.
| * **Broader vLLM compatibility and upstream alignment:** track newer vLLM and vLLM-Ascend releases, evaluate model runner v2, keep compatibility patches minimal, and contribute generally useful abstractions upstream as they mature. | |
| * **Broader vLLM compatibility and upstream alignment:** track newer vLLM releases, evaluate model runner v2, keep compatibility patches minimal, and contribute generally useful abstractions upstream as they mature. |
Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
GindaChen
left a comment
GindaChen
left a comment
There was a problem hiding this comment.
I opened a small polishing PR against this branch: jiangkuaixue123#1. Inline notes below map the proposed edits to the current draft.
|
|
||
| We are excited to introduce [**vLLM AFD Plugin**](https://github.com/vllm-project/afd-plugin), an experimental external plugin that brings **Attention–FFN Disaggregation (AFD)** to vLLM. | ||
|
|
||
| Mixture-of-Experts (MoE) inference combines two very different kinds of work inside every transformer layer. Attention is stateful and closely coupled to request scheduling and the KV cache, while the FFN or expert path is dominated by routed expert computation and all-to-all communication. Serving both paths in one worker topology forces them to share the same scaling, execution, and communication choices. |
There was a problem hiding this comment.
In the polishing PR, I tighten this intro so it ends on the core value proposition: AFD preserves the vLLM request lifecycle and OpenAI-compatible serving surface while allowing the Attention and FFN paths to scale independently. This avoids repeating the problem framing that the next section covers.
| > [!NOTE] | ||
| > This project is still experimental and needs more large-scale testing across different hardware backends. | ||
|
|
||
| ## Why Attention–FFN Disaggregation? |
There was a problem hiding this comment.
The polishing PR keeps the heading wording consistent with the intro by using Attention-FFN Disaggregation. This is just a typography/style consistency change across the post.
| ## Why Attention–FFN Disaggregation? | ||
|
|
||
| MoE serving systems must balance several competing demands: | ||
|
|
There was a problem hiding this comment.
I reframe this paragraph around the challenge rather than the solution. Since the intro already says AFD separates the paths, this section reads more clearly if it explains why a shared worker topology is difficult for MoE serving.
| MoE serving systems must balance several competing demands: | ||
|
|
||
| 1. **Different scaling dimensions.** Attention capacity follows request state, sequence length, and KV-cache pressure. Expert capacity follows token routing and expert load. AFD gives each path its own rank topology instead of requiring one shared layout. | ||
| 2. **Different runtime responsibilities.** Attention needs scheduling, KV-cache coordination, and sampling. FFN execution only needs activations, routing metadata, and a way to return expert outputs. Splitting the services lets the FFN side run as a lightweight connector-driven daemon. |
There was a problem hiding this comment.
The polishing PR changes the lead-in to Making this separation practical requires... so the bullets become concrete system design challenges. That makes the section's role clearer before entering the architecture section.
| 1. **Different scaling dimensions.** Attention capacity follows request state, sequence length, and KV-cache pressure. Expert capacity follows token routing and expert load. AFD gives each path its own rank topology instead of requiring one shared layout. | ||
| 2. **Different runtime responsibilities.** Attention needs scheduling, KV-cache coordination, and sampling. FFN execution only needs activations, routing metadata, and a way to return expert outputs. Splitting the services lets the FFN side run as a lightweight connector-driven daemon. | ||
| 3. **Backend-specific communication.** CUDA and Ascend expose different collective libraries, graph runtimes, and optimized MoE operators. A common connector contract keeps the model-facing flow stable while allowing each backend to own its data path. | ||
| 4. **Room for communication/computation overlap.** Asynchronous dispatch and MoE ubatching can overlap independent stages instead of serializing all expert work behind the Attention path. |
There was a problem hiding this comment.
I adjust this bullet to different scaling requirements and different rank topologies. The intent is the same, but the phrasing is more idiomatic and directly describes the scaling mismatch between Attention and FFN.
|
|
||
| * **Request-driven Attention service.** The Attention worker retains vLLM's scheduler, KV cache, batching, model lifecycle, and sampling path. A plugin-owned model runner installs AFD metadata into the forward context and publishes data-parallel, ubatch, layer, and graph state to the FFN side. | ||
| * **Connector data and control plane.** At each split layer, the model wrapper sends Attention hidden states to the FFN service and receives the computed FFN output. A backend-neutral connector interface carries both tensors and the metadata required to interpret them. | ||
| * **Connector-driven FFN service.** The FFN worker has no request traffic, scheduler, or KV cache. A background loop receives metadata and activations, invokes `compute_ffn_output()` on the plugin-owned model wrapper, and sends the result back to Attention. Requests are always sent to the Attention API server. |
There was a problem hiding this comment.
I simplify the component labels to Attention service, FFN service, and Connector layer. The original labels were accurate, but the shorter labels make the architecture easier to scan.
|
|
||
| This boundary is deliberately narrow. vLLM continues to own the serving control plane where its existing abstractions fit, while the plugin owns the AFD workers, model runners, connectors, metadata, model split points, and a small set of version-scoped compatibility patches. | ||
|
|
||
| ### Connector and backend support |
There was a problem hiding this comment.
The polishing PR rewrites this boundary paragraph as an intentionally small integration surface, and fixes the grammar around the plugin provides. The goal is to make the division of ownership between vLLM and the plugin more explicit.
|
|
||
| The same high-level exchange—Attention output to FFN, FFN output back to Attention—is shared across connectors. Backend packages remain separate so CUDA graph behavior, ACL graph behavior, NCCL communication, and Ascend custom operators do not leak into one another. | ||
|
|
||
| ### Key features |
There was a problem hiding this comment.
I convert the em dash phrasing here to plain hyphen-separated parenthetical text for consistency with the surrounding ASCII style used in the updated draft.
|
|
||
| ### Key features | ||
|
|
||
| * **Native vLLM serving surface.** Existing vLLM users still launch with `vllm serve`, send requests to an OpenAI-compatible endpoint, and configure the runtime through `--additional-config`. |
There was a problem hiding this comment.
The polishing PR renames Key features to Supported features because this section includes both capabilities and explicit current-scope limitations. That title sets reader expectations a little better.
| * **MoE model integration.** The plugin registers wrappers for DeepSeek V2/V3-family architectures, including DeepSeek V3.2, and GLM MoE DSA. The wrapper exposes separate Attention and FFN computations while reusing upstream layer implementations. | ||
| * **Graph and ubatching paths.** The synchronous GPU and NPU connectors support decode-only graph capture. Dual Batch Overlap is supported with exactly two ubatches, and CAM async provides AFD-managed MoE ubatching for its prefill path. | ||
|
|
||
| ## A Performance Snapshot |
There was a problem hiding this comment.
The moved role-pruned model loading note is placed after the supported feature list in the polishing PR. It also remains in a Markdown note block, with the body line quoted correctly so the admonition renders as intended.
|
Thrilled to see the AFD plugin! I opened a polishing PR against the source branch here: jiangkuaixue123#1 That PR contains the wording and structure updates noted in my inline review comments. You should be able to merge that PR directly in here and have these edits reflected back in this PR. Authored: @Viol2000 @yuxuandexter @GindaChen |
Signed-off-by: Junda Chen <32371474+GindaChen@users.noreply.github.com> Signed-off-by: Yichao Fu <57950249+Viol2000@users.noreply.github.com> Signed-off-by: Yuxuan Zhang <116124872+yuxuandexter@users.noreply.github.com> Co-authored-by: Junda Chen <32371474+GindaChen@users.noreply.github.com> Co-authored-by: Yichao Fu <57950249+Viol2000@users.noreply.github.com> Co-authored-by: Yuxuan Zhang <116124872+yuxuandexter@users.noreply.github.com>
21580e1 to
56922ff
Compare
18671f0 to
addcdfb
Compare
6e91f2b to
861b862
Compare
Signed-off-by: ywang96 <ywang96@users.noreply.github.com>
Signed-off-by: ywang96 <ywang96@users.noreply.github.com>
Rename the post and its asset directory from 2026-07-14 to 2026-07-23 and update the asset references accordingly. Signed-off-by: ywang96 <ywang96@users.noreply.github.com>
861b862 to
b788673
Compare
4 participants
Summary
Why
The AFD Plugin repository has detailed implementation and runtime documentation, but the vLLM community does not yet have a concise ecosystem-level introduction to its motivation, architecture, current capabilities, limitations, and future direction.
The source draft is also available in vllm-project/afd-plugin#114 for project-level review.
Review notes
CAMP2pAFDConnectordecode performance subsection is intentionally left as a placeholder until benchmark results are available.vllm-project/afd-plugincanonical location and should be confirmed before publication.Validation
xmllint --noout assets/figures/2026-07-14-vllm-afd-plugin/vllm-afd-plugin-architecture.svggit diff --cached --checkThe local Jekyll build could not be run because the host provides Ruby 2.6 and Bundler 1.17, which cannot resolve the repository's current gem dependencies. The pull request's GitHub Actions build should validate the site with the configured Ruby environment.